SYNCHRONOUS TRANSMISSION OF DATA 
WITH NETWORK REMOTE CONTROL 


BACKGROUND OF THE INVENTION 

The present invention relates to the field of remote control of devices in 
5 networks. In particular, the present invention is concerned with remotely controlling 
the transmission of complementary data over networks. 

The development of networks has led to the development of many beneficial 
network services, especially in the area of communication. However, networks can 
also be and have been used for controlling devices that are connected to other 
10 devices through such networks. For example, in voice mail messaging systems, 
voice mail servers are routinely controlled by phones connected to voice mail 
servers through the Public Switched Telephone Network. 

One area which has lacked sufficient attention in the context of the remote 
control of devices through networks has been in the provision of data services. In 
15 particular, there is a need for devices that can be remotely controlled to transmit 
synchronous, complementary data to a plurality of devices over a network. 

SUMMARY OF THE INVENTION 

A novel method and apparatus is disclosed for synchronously delivering 
complementary data to two devices. The steps in one method embodiment of the 
20 present invention comprise: i) receiving a command through a network for 

synchronous transmission of complementary data, and ii) transmitting first data 
through a network to a first device, the transmission of the first data synchronized 
with transmission of second data transmitted to a second device, the first and second 
data forming complementary data. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 illustrates a system in accordance with embodiments of the present 
invention. 

Figure 2 illustrates a system in accordance with embodiments of the present 
5 invention in which a first device is connected through a first network and a second 
network to a second device. 

Figure 3 illustrates a system in accordance with an embodiment of the 
present invention in which a video program is transmitted to a first device and 
corresponding subtitles are transmitted to a second device. 
10 Figure 4 illustrates a flow diagram of the steps in an embodiment of the 

present invention. 

Figure 5 illustrates a system in accordance with an embodiment of the 
present invention in which a device is used to control a telephone through a network. 

DETAILED DESCRIPTION 

1 5 Embodiments of the present invention allow for the synchronous 

transmission of complementary data to two devices. 

In one embodiment of the invention, a mobile and/or hand-held device such 
as a PDA is used to control another device remotely through one or more networks, 
^uch that complementary, synchronous streams of data that are intended for 

20 observation in combination by users and/or entities, can be transmitted to each of the 
PDA and the device. The flexibility of such an arrangement, wherein the PDA can 
be carried to different locations to control different devices for synchronous 
transmission of complementary data, allows the provision of a multitude of services 
not heretofore available. 

25 ^ Figure 1 shows a system in accordance with embodiments of the invention. 

In Figure 1, a first device 120 and a second device 130 are connected to network 
110. Network 110 may be any type of network, for example, the Internet, an 
intranet, a television network, a wireless network, a public switched telephone 
network, a combination of these networks, or any other network. Moreover, 

30 network 1 10 may include two different networks connected by a gateway; for 
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example, network 110 may include any two of the networks above connected by a 
gateway server allowing for the communication of data and commands from each 
network to the other. 

Device 120 and device 130 are devices which are capable of receiving data 
5 transmitted through network 110. In particular, device 120 and device 130 are 
capable of receiving complementary data. "Complementary data" is used herein to 
designate at least two streams of related data that are semantically related to each 
other such that, when combined, form an organic whole. For example, the audio 
data and the video data for a television program may be considered to be 

10 complementary data. 

One other property of complementary data as used herein is that the at least 
two streams of data are intended for observation by the same entity or person, or are 
actually observed by the same entity or person. For example, a video signal in a 
video program that is observed by many users is complementary to audio data of the 

15 program that is played on each user's individual player (e.g. in different languages). 
However, first computer game data that is streamed to a video monitor of a first user 
is not complementary to second computer game data that is streamed to a video 
monitor for a second user, where the first user is playing the computer game with or 
against the second user, because the first data is intended for observation by the first 

20 user alone, whereas the second data is intended for observation by the second user 
alone. 

Another property of complementary data as used herein is that at least one 
stream of data is not broadcast; it is either unicast to a single user or device, or 
multicast to a predetermined list of users and/or devices. As is known in the art, 

25 unicast and multicast require a predetermined list of users and/or devices to which 
transmission is to be made. Such a list may be as simple as a list of network 
addresses, or may include additional information. 

Additionally, device 120 and 130 are capable of synchronously playing back, 
displaying and/or transmitting streams of data. Two streams of data may be 

30 considered to be synchronously played back, displayed and/or transmitted if the play 
back, display or transmission of each stream has a special temporal relation to that of 
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the other, such that the streams of data in combination are properly perceived by an 
observer, or properly received or processed by a device. For example, the audio data 
and the video data for a television program are played back and displayed, 
respectively, synchronously when the combined data is perceived in the intended 
5 manner by an observer. 

Device 120 and device 130 are additionally capable of processing, and/or 
synchronously displaying or playing back received data. For example, if video data 
is received by a video device (e.g. device 120 and/or device 130), then that device 
may visually display the data or data derived from the received data on a screen. 
□ 10 Alternatively, if audio data is received by an audio-capable device (e.g. device 120 
~1 and/or device 130), then the device may audibly play back the data or data derived 

from the received data through a speaker. Device 120 and device 130 may 
=p additionally include a processor and/or a memory. 

Figure 1 additionally shows that agent 100 is connected to network 110. 
f 15 Agent 100 is capable of receiving commands from a device (e.g. from one or both of 
rfj device 120 and device 130, or a third device) for the transmission of data, and 

jji transmitting a first stream of data to device 120 and a second stream of data to 

y device 130, wherein the first and second streams of data are complementary data. 

Agent 100 in particular is capable of receiving and responding to a command to 
20 transmit a first stream of data to a first device over a network and a second stream of 
data to a second device over the network, wherein the first and second streams of 
data form complementary data. Agent 1 00 may include a processor and a memory 
in addition to a connection or connections to one or more networks. 

Agent 100 may determine the timing of the transmission of data to the first 
25 and second devices such that the data, when displayed and/or played back in real- 
time by the first and second device, is properly synchronized. For example, if the 
first stream of data is video data corresponding to an audiovisual recording (e.g. a 
television program) and the second stream of data is audio data corresponding to the 
audiovisual program, then agent 100 may transmit the first and second streams of 
30 data such that someone in proximity to both the first and second devices observes 
the audiovisual program properly without any lag time between the video data and 
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the audio data. Agent 100, to correctly time the transmission of the first and second 
streams of data for proper observation at the first and second device, may measure 
the respective time delays introduced due to propagation of the data through the 
network to the first and second devices, and compensate by delaying the first and 
5 second streams appropriately. For example, if network 1 10 is the Internet, agent 100 
may use the Packet Internet Groper ("ping") utility (e.g. see Internet Control 
Message Protocol, RFC 792) to send a packet of data to each of the first and second 
devices, and measure the time taken for a response to each ping. Then, one half of 
the difference in the respective delay times is the relative delay that would be 

□ 10 introduced into the transmissions of the data streams to the first and second device. 

^ The relative delay can be compensated by delaying the data stream corresponding to 

!Z? the shorter delay time by the relative delay amount. 

Jp Other methods of synchronizing that are well-known to those skilled in the 

p art may also be used in conjunction with embodiments of the present invention, 

f 15 Alternatively, data received at the first and second device may be stored in 

FU memory and displayed and/or played back from memory. The first and second 

j_n streams of data may be displayed and/or played back in a properly synchronized 

tf fashion in this case as well. For example, agent 100 may transmit timing data to 

each of the first and second devices specifying the beginning time of display and/or 
20 playback from memory of the first and second data streams such that a user will 

observe the display and/or playback of the two streams of data in synchronicity. For 
example, agent 100 may transmit the message "begin playback at 4:00 p.m. Eastern 
Standard Time" to both the first and second devices. The first and second devices 
may then each begin the synchronous display and/or playback of their respective 
25 streams of data at 4:00pm, Eastern Standard Time. 

Alternatively, a user may issue a command to one of the devices (e.g., by 
pressing a button) to begin synchronous display and/or playback of data stored in the 
memories of the first and second devices. Such action may cause that device to 
transmit a notify signal to agent 100. Agent 100 may then determine the relative 
30 delay in the transmission paths to the first and second devices in the manner set forth 
earlier. Then, agent 100 may send a begin display and/or begin playback signal to 
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each of the first and second devices wherein the begin display (or begin playback) 
signal corresponding to the shorter transmission path is delayed by the relative 
delay. If each device begins display and/or playback of the data from memory when 
it receives the begin display (or begin playback) signal from agent 100, then the first 
5 and second signals will be properly synchronized as observed by the user. 

Figure 2 shows a system in accordance with embodiments of the present 
invention in which device 120 is connected to a first network 210 and device 130 is 
connected to a second network 215. Each of network 210 and network 215 may be 
the Internet, an intranet, a television network, a wireless network, a public switched 

10 telephone network, a combination of these networks, or any other network. 

Gateway server 225 is connected to both network 210 and network 215, and is 
capable of receiving commands and/or data from either network for transmission to 
a specified destination, properly formatting received commands and/or data for 
transmission through the other network, and transmitting the properly formatted data 

15 through the other network to the specified destination. Figure 2 shows agent 100 to 
be a part of network 210, although agent 100 could instead be a part of network 215, 
or a part of both network 210 and network 215. Apart from the explicit indication of 
two different networks connected by a gateway server, the system shown in Figure 1 
is similar to that shown in Figure 2. 

20 Figure 3 shows a system in an embodiment of the present invention. In this 

embodiment, device 320 is a video device capable of displaying video data that is 
connected to network 310, which is a network capable of carrying the transmission 
of video data, e.g., the Internet or a television network. Figure 3 additionally shows 
device 330, which is capable of displaying visual, textual data. Device 330 is 

25 connected to network 315, which is capable of carrying the transmission of textual 
data. Gateway 325 is connected to both network 3 10 and network 315, and is 
capable of receiving data and/or commands through at least one of the networks for 
transmission to a specified destination, properly formatting and translating the data 
and/or commands for transmission through the other network, and transmitting the 

30 formatted and translated data and/or commands though the other network to the 
specified destination. Agent 300 is capable of receiving commands from a device 
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(e.g., one or both of device 320 and device 330, or a third device) for the 
transmission of data, and transmitting a first stream of data to device 320 and a 
second stream of data to device 330, wherein the first and second streams of data are 
complementary data. Agent 300 may also include a processor and a memory. 
5 The system shown in figure 3 can in particular be of benefit to an audio- 

impaired user. For example, the user may employ a hand-held device (e.g. device 
330) that is connected through a wireless network (e.g. network 315) to agent 300. 
In this implementation, the user, wishing to see subtitles corresponding to a video 
program being displayed on video device 320, may press a button on device 330 or 
q 10 otherwise issue a command to device 330 for displaying subtitles. Device 330 may 
Cj transmit this request to agent 300. In response, agent 300, which is connected 

iff through gateway server 325 to network 310 (and capable of detecting and tracking 

*p the program being displayed on video device 320), may transmit properly 

p synchronized subtitles corresponding to the program being displayed on video 

s s e 15 device 320, to device 330. In this manner, the user is able to observe the 
FU complementary data that forms the program in a manner that is maximally useful to 

Ifl him/her and not distracting to other viewers of the program being displayed on 

jf device 320. This embodiment may also be used where the program being displayed 

on video device 320 is in a language that is foreign to the user. In this case, agent 
20 300 may transmit synchronized subtitles or audio signals in the user's language 
corresponding to the program, to device 330. 

There are many other contexts in which the systems shown in Figures 1-3 
can be of benefit. For example, a navigation system can be implemented with the 
option of audible output. In this context, device 120 in Figure 1 or 2 is a navigation 
25 system installed in a vehicle that is capable of assisting the operator in navigating the 
vehicle. The navigation system may be connected through, for example, a wireless 
connection to the Internet (e.g. network 1 10 in Figure 1 or network 210 in Figure 2). 
Agent 100, which is connected to device 120 through network 1 10 (or 210), is 
capable of generating and transmitting supplementary information, or providing 
30 supplementary services, that can be used by the operator of the vehicle in connection 
with navigating the vehicle. For example, agent 100 may track the location of the 
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vehicle through transmissions from device 120 and determine the location of 
services near the vehicle; e.g., the location of hospitals, banks, gasoline stations and 
other services in the vicinity of the current position of the vehicle. Agent 100 may 
then transmit this information, either through the same network (network 1 10 of 
5 Figure 1) or through another network (e.g. network 215 of Figure 215) to device 
130, which is in possession of the operator of the vehicle. For example, device 130 
may be a mobile, hand-held portable device (e.g., a personal digital assistant, a 
cellular phone, a pocket computer) that the operator can use even when not present 
inside the vehicle. 

10 Alternatively, device 130 may be a device that receives and plays back 

audible instructions corresponding to directions displayed on the navigation system. 
For example, device 130 may play back instructions such as "turn left at the next 
intersection" that may allow the operator to receive navigational instructions without 
diverting his/her attention from the task of operating the vehicle. 

15 Figure 4 shows an embodiment of the present invention. At step 410, a 

command is received through a network for the synchronous transmission of data. 
For example, agent 100 of figure 1 may receive a command from device 120, device 
130 or a third device for the synchronous transmission of complementary data to 
device 120 and device 130. 

20 At step 420, first data is transmitted to a first device synchronously with 

second data to a second device, wherein the first data and the second data are 
complementary. For example, agent 100 of figure 1 may synchronously transmit 
first data to device 120 and second data to device 130. 

In another aspect of the present invention, the systems shown in Figure 1 and 

25 2 can be used for the remote control of devices without synchronous transmission of 
complementary data. For example, commands transmitted by device 120 through 
network 1 10 in Figure 1 may cause agent 100 to in turn transmit commands to 
control device 130. One example of such a system is shown in Figure 5. In Figure 
5, phone 520 and 525 are connected to the Public Switched Telephone Network 

30 ("PSTN") 550, which contains agent 500. PSTN 550 is connected to gateway server 
540, which is in turn connected to Internet 560. Device 530 is connected to Internet 
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560, for example, through a wireless connection. In this embodiment, device 530 
can transmit commands to cause phone 520, for example, to dial and connect to 
phone 525. In one application of this system, phone 520 is a head-phone that can be 
carried on the head of the operator, and device 530 is a personal digital assistant 
5 ("PDA") containing the address book of the operator. The operator can select an 
entry from the address book in PDA 530, for example, the entry corresponding to the 
telephone number for phone 525, for connection to phone 520. PDA 530 may then 
transmit a command through Internet 560, gateway 540 and PSTN 550 to agent 500 
for connecting phone 520 to phone 525. Agent 500 may then connect phone 520 to 
10 phone 525. In this manner, a telephone may be controlled remotely through a 
network. 

Variations of the systems shown in Figures 1 and 2 can be used to control 
other types of devices through a network in the same manner as just described for 
controlling phones. For example, the system shown in Figure 1 or Figure 2 can be 

15 used to control appliances (e.g., heating, cooling, lighting, music systems and other 
devices) in the same manner as described above, where device 120 is an appliance 
connected to a network of appliances. 

Control of a device may pertain to either or both of i) controlling the setup of 
a communication channel with another device; and, ii) controlling the functionality 

20 of the device, or functionality in connection with the device, after a communication 
channel with another device has been established. 

An advantage of an embodiment of the invention derives from the flexibility 
of a mobile and/or hand-held device such as a PDA being used to control another 
device through one or more networks, such that complementary, synchronous 

25 streams of data, intended for observation in combination by users and/or entities, can 
be transmitted to each of the PDA and the device. 

A method and system have been disclosed for controlling devices through a 
network. In particular, a method and system have been disclosed for remotely 
controlling a device for the synchronous transmission of complementary data to a 

30 second and third device. The present invention has been described in terms of 

several embodiments solely for the purpose of illustration. Persons skilled in the art 
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will recognize from this description that the invention is not limited to the 
embodiments described, but may be practiced with modifications and alterations 
limited only by the spirit and scope of the appended claims. 
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